Skip to content
This repository has been archived by the owner on Feb 20, 2023. It is now read-only.

Issue #19956: Add telemetry for tab view setting changes #19959

Merged
merged 3 commits into from
Aug 6, 2021

Conversation

jonalmeida
Copy link
Contributor

This differs from tab_view_setting which tells us what the user's tab
setting is at startup. It does not tell us if the user explicitly
changed it instead of just using the default (which was recently
changed in #19809).

Pull Request checklist

  • Tests: This PR includes thorough tests or an explanation of why it does not
  • Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not
  • Accessibility: The code in this PR follows accessibility best practices or does not include any user facing features. In addition, it includes a screenshot of a successful accessibility scan to ensure no new defects are added to the product.

To download an APK when reviewing a PR:

  1. click on Show All Checks,
  2. click Details next to "Taskcluster (pull_request)" after it appears and then finishes with a green checkmark,
  3. click on the "Fenix - assemble" task, then click "Run Artifacts".
  4. the APK links should be on the left side of the screen, named for each CPU architecture

@codecov-commenter
Copy link

codecov-commenter commented Jun 11, 2021

Codecov Report

❗ No coverage uploaded for pull request base (main@01555c2). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 61b0253 differs from pull request most recent head e1ef3c9. Consider uploading reports for the commit e1ef3c9 to get more accurate results
Impacted file tree graph

@@           Coverage Diff           @@
##             main   #19959   +/-   ##
=======================================
  Coverage        ?   36.51%           
  Complexity      ?     1634           
=======================================
  Files           ?      543           
  Lines           ?    21214           
  Branches        ?     3176           
=======================================
  Hits            ?     7746           
  Misses          ?    12590           
  Partials        ?      878           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01555c2...e1ef3c9. Read the comment docs.

@jonalmeida
Copy link
Contributor Author

Request for data collection review

  1. What questions will you answer with this data?

    • This data will tell us if a user changes the tab tray view when the (new) default is grid view.
  2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

    • Yes, this gives us an understanding of user experience if our new default is the right one and we increase in usage.
  3. What alternative methods did you consider to answer these questions? Why were they not sufficient?

    • No alternative is available to answer this question.
  4. Can current instrumentation answer these questions?

    • No, we have tab_view_setting that exists, but this can incorrectly skew our data for users that have not changed the default settings and will see the new default.
  5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Description Data Collection Category Tracking Bug #
Tab view setting is list or grid category 2 - Interaction data #19956
  1. How long will this data be collected? Choose one of the following:

    • I want this data to be collected for 1 year initially (ending on May 10, 2022), with the option to renew at that point.

  2. What populations will you measure?

    • No filters - all locales, channels, etc.
  3. If this data collection is default on, what is the opt-out mechanism for users?

    • Default Glean SDK opt-out mechanism.
  4. Please provide a general description of how you will analyze this data.

    • When we see that our new default increases in usage, and users are not switching away from the new default.
  5. Where do you intend to share the results of your analysis?

    • Internally in the Fenix teams.
  6. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?

    • No third-party tool will use this data.

@gabrielluong
Copy link
Member

gabrielluong commented Jun 11, 2021

We have a similar telemetry for this in "tab_view_setting". I am not quite sure if it's worthwhile to collect the "change" since we already collect a metrics ping about what the tab view is set to everyday. You will also want to consult with DS in #data-help.

Of course, I see you talk about "tab_view_setting" after I wrote all of the above.

@jonalmeida jonalmeida added the needs:data-review PR is awaiting a data review label Jul 9, 2021
@jonalmeida jonalmeida marked this pull request as ready for review July 9, 2021 20:04
@jonalmeida jonalmeida requested review from a team as code owners July 9, 2021 20:04
@travis79
Copy link
Member

Request for data collection review

1. What questions will you answer with this data?
   
   * This data will tell us if a user changes the tab tray view when the (new) default is grid view.

2. Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?
   
   * Yes, this gives us an understanding of user experience if our new default is the right one and we increase in usage.

3. What alternative methods did you consider to answer these questions? Why were they not sufficient?
   
   * No alternative is available to answer this question.

4. Can current instrumentation answer these questions?
   
   * No, we have `tab_view_setting` that exists, but this can incorrectly skew our data for users that have not changed the default settings and will see the new default.

5. List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories found on the Mozilla wiki.

Measurement Description Data Collection Category Tracking Bug #
Tab view setting is list or grid category 2 - Interaction data #19956

1. How long will this data be collected? Choose one of the following:
   
   * I want this data to be collected for 1 year initially (ending on May 10, 2022), with the option to renew at that point.

2. What populations will you measure?
   
   * No filters - all locales, channels, etc.

3. If this data collection is default on, what is the opt-out mechanism for users?
   
   * Default Glean SDK opt-out mechanism.

4. Please provide a general description of how you will analyze this data.
   
   * When we see that our new default increases in usage, and users are not switching away from the new default.

5. Where do you intend to share the results of your analysis?
   
   * Internally in the Fenix teams.

6. Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection?
   
   * No third-party tool will use this data.

Data Review

  1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, through the metrics.yaml file and the Glean Dictionary

  1. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, through the "Send Usage Data" toggle in the app preferences

  1. If the request is for permanent data collection, is there someone who will monitor the data over time?

N/A, collection set to end May, 2022

  1. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, User Interaction Data

  1. Is the data collection request for default-on or default-off?

Default-on

  1. Does the instrumentation include the addition of any new identifiers?

No

  1. Is the data collection covered by the existing Firefox privacy notice?

Yes

  1. Does the data collection use a third-party collection tool?

No

Result

data-review+

@travis79 travis79 removed the needs:data-review PR is awaiting a data review label Jul 19, 2021
@mergify
Copy link
Contributor

mergify bot commented Jul 28, 2021

This pull request has conflicts when rebasing. Could you fix it @jonalmeida? 🙏

@jonalmeida jonalmeida added the pr:needs-landing PRs that are ready to land [Will be merged by Mergify] label Aug 6, 2021
This differs from `tab_view_setting` which tells us what the user's tab
setting is at startup. It does not tell us if the user explicitly
changed it instead of just using the default (which was recently
changed in mozilla-mobile#19809).
@jonalmeida
Copy link
Contributor Author

Intermittent test failing from mockk. I'll fix this in a separate PR and restart the build here until then:

AddonsManagementViewTest. onAddonItemClicked shows details if addon is not installed
io.mockk.MockKException: Can't instantiate proxy for class kotlin.Function1
	at io.mockk.impl.instantiation.JvmMockFactory.newProxy(JvmMockFactory.kt:64)
	at io.mockk.impl.instantiation.AbstractMockFactory.newProxy$default(AbstractMockFactory.kt:29)
	at io.mockk.impl.instantiation.AbstractMockFactory.mockk(AbstractMockFactory.kt:59)
	at io.mockk.impl.annotations.JvmMockInitializer.assignRelaxedMockK(JvmMockInitializer.kt:142)
	at io.mockk.impl.annotations.JvmMockInitializer.initMock(JvmMockInitializer.kt:47)

@jonalmeida jonalmeida closed this Aug 6, 2021
@jonalmeida jonalmeida reopened this Aug 6, 2021
@mergify mergify bot merged commit 3108107 into mozilla-mobile:main Aug 6, 2021
@jonalmeida jonalmeida deleted the issue-19956 branch August 6, 2021 19:58
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
pr:needs-landing PRs that are ready to land [Will be merged by Mergify]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants